Scanning display systems with photonic integrated circuits

ABSTRACT

A display system may display image frames. The system may include multiple sets of laser dies. Each set of laser dies may emit a respective set of beams of light to a photonic integrated circuit. Each set of beams may include light in at least three wavelength ranges that include visible and/or infrared wavelengths. Channels in the photonic integrated circuit may receive the sets of beams with a first pitch and may emit the set of beams with a second pitch that is finer than the first pitch and at a given angular separation to tangential and sagittal axis scanning mirrors.

This application claims the benefit of provisional application No.62/731,448, filed Sep. 14, 2018, which is hereby incorporated byreference herein in its entirety.

BACKGROUND

This relates generally to display systems, including display systemscontaining microprojectors and displays.

Electronic devices and other systems often include displays. Forexample, a head-mounted device such as a pair of virtual reality ormixed reality glasses may have a display for displaying images for auser, or a projector system may include a projector for projecting lightfields to a display. The projector may include light sources that emitlight fields and an ancillary optical system that conveys the emittedlight to the user.

It is challenging to form a projection and display system withsufficient optical brightness, display resolution, and compactness for ascalable use. Additional care must be taken to consider user cases thatinclude drop shock and thermal loads.

SUMMARY

A display system such as a display system in an electronic device maydisplay an image frame. The display system may include a scanning mirrorand an array of staggered light emitting elements arranged in diagonalrows and aligned vertical columns. The staggered light emitting elementsmay emit light fields that propagate to a scanning via a lens. Thescanning mirror may reflect the image light while rotating about anaxis. Control circuitry may selectively activate the light emittingelements across tangential and sagittal axes of the image frame (e.g.,using selected timing delays) while controlling the scanning mirror toscan across the sagittal axis at a scanning frequency. This mayconfigure the reflected image light from the scanning mirror to appearas a continuous column of pixels from the image frame (e.g., continuouscolumns across the entire two-dimensional image frame).

Additional arrays of light emitting element staggered along the sagittalaxis may be used if desired. The additional arrays may have a largerangular spacing to perform foveated imaging operations if desired. Theadditional arrays may include infrared emitters, optical emitters,and/or sensors to perform gaze tracking. The additional arrays may emitlight of other colors.

In another suitable arrangement, the display system may include a fastscanning mirror and a slow scanning mirror that scan along tangentialand sagittal axes of the projected display. That fast scanning mirrormay receive multiple beams of light from a photonic integrated circuit.The photonic integrated circuit may receive light from multiple sets oflaser dies (e.g., laser dies driven by a corresponding laser driver).The photonic integrated circuit may include channels that convey thelight from the sets of laser dies and that emit the light as themultiple beams provided to the fast scanning mirror. The channels mayhave a wide pitch to accommodate the relatively large size of the laserdies and may reduce the pitch before emitting the beams to maximizeresolution of the displayed image frame. The fast and slow scanningmirrors may fill the image frame with the beams of light.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an illustrative system that may include a displaysystem in accordance with an embodiment.

FIG. 2 is a top-down view of illustrative optical system components thatinclude a scanning mirror for providing image light to a user inaccordance with an embodiment.

FIG. 3 is a cross-sectional side view of illustrative optical systemcomponents of the type shown in FIG. 2 in accordance with an embodiment.

FIG. 4 is a perspective view showing how illustrative optical systemcomponents of the type shown in FIGS. 2 and 3 may be mounted within anarrow optical system housing in accordance with an embodiment.

FIGS. 5 and 6 are front views of illustrative arrays of light sourceshaving staggered light emitting elements in accordance with anembodiment.

FIG. 7 is a diagram showing how an image displayed by an illustrativedisplay may include a high definition foveated region surrounded by alower resolution region in accordance with an embodiment.

FIG. 8 is a front view showing how illustrative light source controlcircuitry and interconnected circuitry may be formed between arrays oflight sources in accordance with an embodiment.

FIG. 9 is a front view of an illustrative array of light sources havingstaggered light emitting elements and microlenses overlapping each ofthe light emitting elements in accordance with an embodiment.

FIG. 10 is a rear view of an illustrative array of light sources havingcontact pads overlapping the center of each light source in the arrayand having staggered light emitting elements in accordance with anembodiment.

FIG. 11 is a cross-sectional side view of illustrative arrays of lightsources of the types shown in FIGS. 5-10 in accordance with anembodiment.

FIG. 12 is a top-down view of illustrative optical system componentsthat include two independent scanning mirrors and a photonic integratedcircuit for providing image light to a user in accordance with anembodiment.

FIG. 13 is a perspective view of illustrative light sources for opticalsystem components of the type shown in FIG. 12 in accordance with anembodiment.

FIG. 14 is a top-down view of illustrative optical system componentsthat include two independent scanning mirrors and a photonic integratedcircuit coupled to light sources over optical fibers for providing imagelight to a user in accordance with an embodiment.

FIG. 15 is a graph showing how image light may be scanned to fill afield of view using illustrative optical system components of the typeshown in FIGS. 12-14 in accordance with an embodiment.

DETAILED DESCRIPTION

Display systems may be integrated into electronic devices such ashead-mounted devices or other electronic devices used for virtualreality and mixed reality (augmented reality) systems. These devices mayinclude portable consumer electronics (e.g., portable electronic devicessuch as cellular telephones, tablet computers, glasses, other wearableequipment), head-up displays in cockpits, vehicles, etc., display-basedequipment (projectors, televisions, etc.). The display system or thedevice in which the display system is located may include a projectionsystem that may be used in other implementations where far-fieldprojection of a light field is necessary. This may include, but is notlimited to, wearable ocular devices, home theater applications, andvirtual/mixed/augmented reality devices. The projection system describedherein contains high resolution and foveated scanning capabilities thatare flexible. These examples are, however, merely illustrative. Anysuitable equipment may be used in providing a user or users with visualcontent using the display systems described herein.

A head-mounted device such as a pair of augmented reality glasses thatis worn on the head of a user may be used to provide a user withcomputer-generated content that is overlaid on top of real-worldcontent. The real-world content may be viewed directly by a user througha transparent portion of an optical system. The optical system may beused to route images from one or more pixel arrays in a display systemto the eyes of a user. A waveguide such as a thin planar waveguideformed from a sheet of transparent material such as glass or plastic orother light guide may be included in the optical system to convey imagelight from the pixel arrays to the user. The display system may includereflective displays such as liquid-crystal-on-silicon displays,microelectromechanical systems (MEMs) displays (sometimes referred to asdigital micromirror devices), or other displays.

A schematic diagram of an illustrative system that may be provided witha display system is shown in FIG. 1. As shown in FIG. 1, system 10 maybe a home theater system, television system, head-mounted device,electronic device, or other system for projecting far field image light.System 10 may include support structures such as support structure 20.In scenarios where system 10 is a head-mounted device, support structure20 may include head-mounted support structures. The components of system10 may be supported by support structure 20. Support structure 20, whichmay sometimes be referred to as a housing or case, may be configured toform a frame of a pair of glasses (e.g., left and right temples andother frame members), may be configured to form a helmet, may beconfigured to form a pair of goggles, or may have other head-mountableconfigurations, or may be configured to form any other desired housingstructures for some or all of the components in system 10.

The operation of system 10 may be controlled using control circuitry 16.Control circuitry 16 may include storage and processing circuitry forcontrolling the operation of system 10. Circuitry 16 may include storagesuch as hard disk drive storage, nonvolatile memory (e.g.,electrically-programmable-read-only memory configured to form a solidstate drive), volatile memory (e.g., static or dynamicrandom-access-memory), etc. Processing circuitry in control circuitry 16may be based on one or more microprocessors, microcontrollers, digitalsignal processors, baseband processors, power management units, audiochips, graphics processing units, application specific integratedcircuits, and other integrated circuits. Software code may be stored onstorage in circuitry 16 and run on processing circuitry in circuitry 16to implement operations for system 10 (e.g., data gathering operations,operations involving the adjustment of components using control signals,image rendering operations to produce image content to be displayed fora user, etc.).

System 10 may include input-output circuitry such as input-outputdevices 12. Input-output devices 12 may be used to allow data to bereceived by system 10 from external equipment (e.g., a tetheredcomputer, a portable device such as a handheld device or laptopcomputer, or other electrical equipment) and to allow a user to providesystem 10 with user input. Input-output devices 12 may also be used togather information on the environment in which system 10 is operating.Output components in devices 12 may allow system 10 to provide a userwith output and may be used to communicate with external electricalequipment. Input-output devices 12 may include sensors and othercomponents 18 (e.g., image sensors for gathering images of real-worldobject that are digitally merged with virtual objects on a display insystem 10, accelerometers, depth sensors, light sensors, haptic outputdevices, speakers, batteries, wireless communications circuits forcommunicating between system 10 and external electronic equipment,etc.).

As shown in FIG. 1, input-output devices 12 may include one or moredisplays in a display system such as display system 14. Display system14, which may sometimes be referred to as a display or light engine, maybe used to display images for a user or users of system 10. Displaysystem 14 include light sources such as light sources 14A that produceillumination (image light) 22. Illumination 22 may pass through opticalsystem 14B. Light sources 14A may include arrays of light sources havinglight emitting elements (e.g., pixels). Optical system 14B may includeone or more scanning mirrors that scan the light emitted by the lightemitting elements (light 22) towards one or more users at location 24for viewing. In scenarios where system 10 is implemented on ahead-mounted device, location 24 may be an eye box, for example. Inscenarios where system 10 is implemented in other types of display-baseddevices, location 10 may be a projector screen or other display screen,a wall, or any other desired far-field location. Selectively activatingthe light elements in light sources 14A and scanning the mirrors acrossone or two dimensions (axes) may allow two-dimensional images to beprojected at location 24 (e.g., across a field of view of at location24).

Optical system 14B may include other optical components such as prisms,additional mirrors, beam splitters, holograms, gratings (e.g.,electrically tunable gratings), lenses, waveguides, polarizers, and/orother optical components to convey light 22 to location 24. If desired,system 14B may contain components (e.g., an optical combiner, etc.) toallow real-world image light 26 (e.g., real-world images or real-worldobjects such as real-world object 28) to be combined optically withvirtual (computer-generated) images such as virtual images in imagelight 22. In this type of system, which is sometimes referred to as anaugmented reality system, a user of system 10 may view both real-worldcontent and computer-generated content that is overlaid on top of thereal-world content. Camera-based augmented reality systems may also beused in system 10 (e.g., in an arrangement which a camera capturesreal-world images of object 28 and this content is digitally merged withvirtual content on display system 14). Display system 14 may be used ina virtual reality system (e.g., a system without merged real-worldcontent) and/or any suitable type of system for projecting light to adesired display location.

In general, it is desirable to provide the scanning mirrors in opticalsystem 14B as large a reflective area as possible (e.g., 1.2-2.0 mm indiameter) in order to maximize image resolution in the far-field domain.Smaller mirrors, for example, generate greater light divergence and thuslower resolution in the far-field than larger mirrors. At the same time,the scanning mirrors need to scan (rotate) at a relatively high speed(frequency) to allow the images to be displayed with a suitably highframe rate. However, scanning the mirrors at high speeds can causephysical deformation in the mirror (particularly for relatively largemirrors), which serves to diverge the image light and thus limit imageresolution in the far-field. It would therefore be desirable to be ableto provide optical systems 14B that can overcome these difficulties toprovide high resolution (low divergence) images at high frame rateswhile still allowing the optical system to fit within the constrainedform factor of system 10.

FIG. 2 is a top-down view of display system 14 in an illustrativeconfiguration in which optical system 14B includes a single scanningmirror. As shown in FIG. 2, optical system 14B may include lenses 40 andscanning mirror 42. Light sources 14A emit image light (light fields) 22that propagate to scanning mirror 42 via lenses 40. Scanning mirror 42rotates about axis 30 (extending parallel to the Z-axis of FIG. 2), asshown by arrows 32. Scanning mirror 42 may include a motor, actuators,MEMS structures, or any other desired structures that control mirror 42to rotate around axis 30 with a desired scanning frequency. Scanningmirror 42 reflects light 22 towards other optical components 14C indisplay 14 and provides coverage over a corresponding field of view 38as the mirror is scanned (rotated) about axis 30. Field of view 38 maybe 45 degrees, between 40 and 50 degrees, less than 40 degrees, greaterthan 45 degrees, or any other desired field of view.

Other optical components 14C may include projection optics (e.g.,optical components for projecting the image light on a display screen,projector screen, wall, or any other desired location 24 as shown inFIG. 1), lenses, beam splitters, optical couplers (e.g., input couplers,output couplers, cross-couplers, etc.), prisms, additional mirrors,holograms, gratings (e.g., electrically tunable gratings), waveguides,polarizers, and/or other optical components to convey light 22 tolocation 24 (FIG. 1). If desired, light sources 14A and optical system14B may be mounted within an optical system housing such as housing 34.Housing 34 may include metal, dielectric, or other materials and mayprotect optical system 14B from misalignment, stray light, dust, orother contaminants. Housing 34 may include a window 36 that passes light22 after being reflected off of scanning mirror 42.

In order to minimize far field light divergence and thus maximizefar-field image resolution, scanning mirror 42 may rotate at arelatively low speed such as the frame rate with which images (imageframes) are displayed using display system 14 (e.g., the frame rate oflight sources 14A). As an example, scanning mirror 42 may rotate at 60Hz, 120 Hz, 24 Hz, between 24 and 120 Hz, between 120 Hz and 240 Hz,greater than 120 Hz, or any other desired frequency. Because scanningmirror 42 rotates at a relatively low speed, scanning mirror 42 mayperform scanning operations without causing significant mechanicaldeformation to the mirror, thereby maximizing mirror size (withoutintroducing excessive deformation), minimizing far-field lightdivergence, and maximizing image resolution in the far-field. Thishorizontal scanning performed by mirror 42 (e.g., in the direction ofarrows 32 around axis 30) may cause light 22 to fill (paint) onedimension of a two-dimensional image frame in the far field that isdisplayed using display system 14. This dimension is sometimes referredto herein as the sagittal axis of display system 14 or the sagittal axisof the displayed image. The axis orthogonal to the sagittal axis is usedto fill (paint) the remainder of the two-dimensional image frame withimage light and is sometimes referred to herein as the tangential axisof display system 14 or the tangential axis of the displayed image(e.g., parallel to the Z-axis of FIG. 2).

In some scenarios, an additional mirror is used to scan over thetangential axis. These tangential axis mirrors rotate at a higherfrequency and can limit far-field image resolution. By omitting anadditional mirror for covering the tangential axis in the example ofFIG. 2, far-field image resolution may be maximized. In the example ofFIG. 2, light sources 14A may include one or more arrays of lightsources arranged in rows and columns. There may be significantly morerows M than columns N in each of the arrays (e.g., M may be at least 10,20, 30, 40, 50, 100, 1000, etc. times N). Each array may therefore haveinsufficient width (parallel to the Y-axis of FIG. 2) to cover theentire sagittal axis of display system 14B, but has sufficient height(parallel to the Z-axis of FIG. 2) to cover the entire tangential axisof display system 14B. Arrays of this type may sometimes be referred toherein as 1.5 dimensional arrays or 1.5D arrays. Arrays that have lessthan 10 times the number of rows as columns may sometimes be referred toherein as 2D arrays. Arrays that have one row or one column maysometimes be referred to herein as 1D arrays.

Light sources 14A may include multiple 1.5D arrays (e.g., separatearrays for emitting different colors such as red (R), green (G), andblue (B) arrays). Control circuitry 16 (FIG. 1) may selectively activate(e.g., turn on/off) the light sources in each 1.5D array with a suitabletiming scheme to fill (paint) the tangential axis of display system 14Bwith far-field light of each color. When combined with the sagittal axisrotation of scanning mirror 42, display system 14 may producetwo-dimensional far-field images at a high frame rate and havingmultiple colors with maximal resolution.

FIG. 3 is a cross-sectional side view of display system 14 of FIG. 2(e.g., as taken in the direction of arrow 41 of FIG. 2). As shown inFIG. 3, light sources 14A (e.g., the 1.5D arrays) have a greater height(parallel to the Z-axis) than width (parallel to the Y-axis) forcovering the tangential axis of the display system. Output collimatingoptics 40 (e.g., multiple lenses) may collimate light 22 (e.g., ˜0.4-1mRad) from the arrays and may direct the light at scanning mirror 42.Scanning mirror 42 rotates around axis 30, as shown by arrows 32, tocover the sagittal axis. The selective activation of light sourcesacross the 1.5D arrays in light sources 14A in combination with rotationof scanning mirror 42 allow light 22 to form a two-dimensional far-fieldimage frame. In the example of FIG. 3, optical system housing 34 isshown as only enclosing lenses 40. This is merely illustrative and, ifdesired, housing 34 may also surround light sources 14A and/or scanningmirror 42 (e.g., as shown in FIG. 2). Lenses 40 may, for example, besplit lenses (sometimes referred to herein as chopped optics or choppedlenses) that are cut in one or more dimensions rather than exhibiting acircular profile in the Y-Z plane (e.g., to help fit the lenses within arelatively narrow housing 34). Because the array of emitters are longerin one dimension (e.g., a non-scanning direction), lenses 40 may includechopped optics to help conserve space without affecting image quality.

FIG. 4 is a perspective view of display system 14 of FIGS. 2 and 3. Asshown in FIG. 4, optical system housing 34 has a window 36. Scanningmirror 42 in housing 34 reflects light 22 through window 36, as shown byarrow 50 (e.g., to other optical components 14C of FIG. 2). Housing 34may have a length 52, a width 56, and a height 54. Length 52 may begreater than height 54 and height 54 may be greater than width 56. As anexample, length 52 may be 30-40 mm, 20-50 mm, 32-38 mm, greater than 50mm, less than 20 mm, or any other desired length. Width 56 may be 4-6mm, 3-10 mm, 5-20 mm, greater than 20 mm, or any other desired width.Height 54 may be 15-20 mm, 10-25 mm, 15-30 mm, greater than 30 mm, lessthan 15 mm, or any other desired height. In this way, housing 34, lightsources 14A, and optical system 14B may exhibit a relatively narrowprofile. This may allow display 14 to be integrated within systems 10having relatively narrow profiles such as head mounted devices (e.g.,within the temple of a glasses frame, helmet, goggles, etc.) or otherminiaturized or portable display systems.

FIG. 5 is a front view of light sources 14A in display 14 (e.g., astaken in the direction of arrow 44 of FIG. 2). As shown in FIG. 5, lightsources 14A may include a light projector such as light projector 70.Light projector 70 includes one or more 1.5D arrays 60 of light sources76 such as 1.5D arrays 60A, 60B, and 60C. The light sources 76 in eacharray 60 (sometimes referred to herein as light cells 76, light sourcecells 76, cells 76, or unit cells 76) may be arranged in a rectangulargrid pattern having rows and columns. Because each array 60 is a 1.5Darray, there are significantly more rows of cells 76 (e.g., extendingparallel to the Y-axis) than columns of cells 76 (e.g., extendingparallel to the Z-axis) in each array.

Each cell 76 may include a corresponding light emitting element 74formed on an underlying substrate 75 (e.g., an array substrate such as asemiconductor substrate). Light emitting elements 74 may include lightemitting diodes (LEDs), organic light emitting diodes (OLEDs), resonantcavity light emitting diodes (RCLEDs), micro light emitting diodes(μLEDs), lasers (e.g., vertical cavity surface emitting lasers(VCSELs)), or any other desired light emitting components. Differentarrays 60 in projector 70 may include different types of light emittingelements 74 (e.g., one array 60 may include RCLEDs whereas another array60 includes VCSELs, etc.). This may allow light any desired color to beemitted by projector 70 (e.g., in scenarios where a single type of lightemitting element is not capable of producing light of a particulardesired wavelength). Light emitting elements 74 may sometimes bereferred to herein as pixels 74.

Each light emitting element 74 in each array 60 may emit light of acorresponding color (wavelength). As one example, the light emittingelements 74 in array 60A may emit red light (e.g., light at a wavelengthbetween 625 nm and 740 nm such as 640 nm), the light emitting elements74 in array 60B may emit green light (e.g., light at a wavelengthbetween 495 nm and 570 nm such as 510 nm), and the light emittingelements 74 in array 60C may emit blue light (e.g., light at awavelength between 420 nm and 495 nm such as 440 nm). In general, arrays60 may emit light at any desired wavelengths (e.g., near-infrared light,visible light, infrared light, ultraviolet light, etc.).

If desired, one or more lower-resolution arrays such as low resolutionarrays 62 (e.g., a first array 62A, a second array 62B, and a thirdarray 62C) may be formed around the periphery of arrays 60 (e.g., at oneor more sides of arrays 60). Low resolution arrays 62 may each includeone or more columns and two or more rows of cells 76. Low resolutionarrays 62 may include larger cells 76 than arrays 60 and light emittingelements 74 in arrays 62 may be spaced farther apart (e.g., providedwith a greater pitch) than light emitting elements 74 in arrays 60(e.g., arrays 62 may exhibit larger angular spreading than arrays 60).Low resolution arrays 62 may be, for example, 1D arrays, 1.5D arrays,and/or 2D arrays. The light emitting elements 74 in each array 62 may bethe same color as the light emitting elements 74 in the adjacent array60 (e.g., light emitting elements 74 in array 62A may emit the samewavelength of light as array 60A, light emitting elements 74 in array62B may emit the same wavelength of light as array 60B, etc.).

Low resolution arrays 62 may generate portions of the image frame in thefar field with greater angular spreading than arrays 60. This may allowfor foveation techniques to be performed on the display images in whicha central portion of the displayed image is provided at higherresolution than peripheral portions of the displayed image. This may,for example, mimic the natural response of the user's eye such that thedisplayed images still appear naturally to the user while also reducingthe resources and data rate required to display the images. Foveationoperations may also be performed by dynamically controlling the speed ofscanning mirror 42. For example, control circuitry 16 may controlscanning mirror 42 to spend more time within the center of the imageframe (e.g., by rotating more slowly through the center of the frame)and less time around the periphery of the image frame (e.g., by rotatingmore rapidly at the periphery of the frame), thereby maximizing imageresolution near the center of the frame while sacrificing imageresolution near the periphery of the frame.

If desired, depth sensing and/or pupil tracking circuitry may beincluded in light sources 14A. In the example of FIG. 5, sources 14Ainclude depth sensing and pupil tracking components 68. Components 68may include an array 64 of infrared light sources 80. Each infrared (IR)light source 80 may include a corresponding infrared light emittingelement 78 (e.g., an IR LED, an IR μLED, an IR VCSEL, etc.). Lightsources 80 emit infrared light that is conveyed towards location 24 byoptical system 14B (FIG. 1). The infrared light reflects off of theuser's eye back towards light sources 14A through optical system 14B.

Components 68 of FIG. 5 may include an array 66 of infrared lightsensors 82. Each infrared light sensor 82 may include a correspondinginfrared light sensitive element 84 (e.g., an IR photodiode, an IRavalanche diode, etc.). Infrared light sensitive element 84 may sensethe reflected infrared light and may provide corresponding infraredimage signals to control circuitry 16. Control circuitry 16 may processthe transmitted and received infrared signals to track the location ofthe user's retina (pupil) at location 24 (e.g., within an eye box), thedirection of the user's gaze, to perform depth sensing, and/or toperform any other desired operations based on the transmitted andreceived infrared signals. Array 64 may include any desired number ofcells 80 arranged in any desired pattern (e.g., array 64 may be a 1.5Darray, a 1D array, a 2D array, etc.). Array 66 may include any desirednumber of cells 82 arranged in any desired pattern (e.g., array 66 maybe a 1.5D array, a 1D array, a 2D array, etc.).

If desired, light sources 14A may perform wavelength multiplexing usingan additional projector such as projector 72 of FIG. 5. Projector 72 mayinclude a 1.5D array 60 for each array 60 in projector 70 (e.g., whereeach array 60 in projector 72 includes the same number and pattern ofcells 76 as the corresponding array 60 in projector 70). In the exampleof FIG. 5, projector 72 includes 1.5D arrays 60 such as arrays 60A′,60B′, and 60C′. Light emitting elements 74 in array 60A′ may emit lightthat is offset in wavelength from the wavelength emitted by array 60A bya predetermined margin (e.g., 20 nm, 30 nm, 40 nm, between 20 and 50 nm,between 10 and 60 nm, etc.). Similarly, array 60B′ may emit light thatis offset in wavelength from the wavelength emitted by array 60B andarray 60C′ may emit light that is offset in wavelength from thewavelength emitted by array 60C by the predetermined margin (e.g., array60A′ may emit 670 nm red light, array 60B′ may emit 540 nm green light,and array 60C′ may emit 470 nm blue light).

Using an additional wavelength-offset projector such as projector 72 mayallow display 14 to perform two different operations on the lightemitted by light sources 14A. For example, other optical components 14C(FIG. 2) may include holograms, diffraction gratings, or otherstructures that are tuned to operate on light of a particularwavelength. Optical components 14C may, for example, include a firsthologram that operates on (e.g., diffracts in a first direction) thewavelengths of light produced by projector 70 and a second hologram thatoperates on the (e.g., diffracts in a second direction) the wavelengthsof light produced by projector 72. This may, for example, allow display14 to display RGB image light that is transmitted to different locationswithin system 10 using the same physically-narrow light sources 14A.

The example of FIG. 5 is merely illustrative. In general, any desirednumber of projectors may be formed within light sources 14A. Any desirednumber of arrays 60 may be formed within each projector. Each array 60may include any desired number of cells 76 arranged in any desiredpattern. Each array 62 may include any desired number of cells 76arranged in any desired pattern. One or more of the arrays 60 and/or 62of FIG. 5 may be omitted. Projector 72 of FIG. 5 may be omitted ifdesired.

As shown in FIG. 5, the light emitting elements 74 within each array 60are horizontally aligned with respect to the light emitting elements 74in the same column of cells 76 but are vertically staggered with respectto the light emitting elements 74 in the same row of cells 76. Forexample, the light emitting elements 74 in each column may be located atdifferent positions along the Z-axis (tangential axis) from the previouscolumn of cells 76 and the next column of cells 76. In other words,light emitting elements 74 are vertically staggered within each array 60(e.g., light emitting elements 74 in each array 60 collectively form astaggered array of light emitting elements having vertical columns anddiagonal rows). If desired, there may be multiple copies of each of thelight emitting elements 74 shown in each diagonal row of FIG. 5 (e.g.,to increase brightness and/or dynamic range relative to scenarios whereeach column of the diagonal row includes only one light emitting element74). Staggering light emitting elements 74 in this way may allow arrays60 to exhibit a fine vertical pitch such that light elements 74 can fillthe tangential axis of display system 14 (e.g., parallel to the Z-axisof FIG. 5) with light even though only a single scanning mirror is usedto scan parallel to the Y-axis (sagittal axis). This may further serveto maximize the resolution of the projected far-field image.

FIG. 6 is a front-view of a given array 60 showing how light emittingelements 74 may be staggered in the physical domain while beingvertically continuous in the optical domain due to the rotation ofscanning mirror 42. The left side of FIG. 6 illustrates the physicaldomain of an exemplary four-by-twelve cell portion of a given array 60(e.g., as taken in the direction of arrow 44 of FIG. 2). The right sideof FIG. 6 illustrates the optical domain of the light emitted by thefour-by-twelve cell portion of array 60 (e.g., as viewed from opticalcomponents 14C of FIG. 2 after reflection by scanning mirror 42).

As shown in FIG. 6, each light emitting element 74 may be formed insubstrate 75 of array 60 and has a corresponding pixel size (opticalpitch) 110. Each light emitting element 74 in the even-numbered columnsmay be vertically offset from the light emitting element 74 immediatelyto its left and the light emitting element 74 immediately to its rightby pixel offset 114. In the example of FIG. 6, the light emittingelement at position (1,2) is vertically offset from the light emittingelement at position (1,1) by pixel offset 114. Similarly, the lightemitting element at position (1,3) is vertically offset from the lightemitting element at position (1,2) by pixel offset 114 and is verticallyoffset from the light emitting element at position (1,1) by offset 115(e.g., twice pixel offset 114). Offset 114 may separate the lightemitting element at position (1,2) from the light emitting element atposition (1,1) and the light emitting element at position (1,3) byrelatively small distance (pixel pitch) along the Z-axis (e.g., within4-6 microns, between 3 and 10 microns, between 3 and 6 microns, etc.).This pattern may repeat one or more times across width 122 of array 60for each row.

In this way, each row includes one or more (repeating) sets 120 of cells76 that include staggered light emitting elements 74 extending acrossthe entire physical row of cells 76 (e.g., parallel to the Z-axis ofFIG. 6). The sum of the pixel pitches 114 in each set 120 may, forexample, be approximately equal to the distance from the bottom edge tothe top edge of the corresponding row (e.g., minus the pixel pitchbetween the light emitting elements in the set). Each row may includeany desired number of sets 120. Each set 120 may include any desirednumber of cells 76 (e.g., three cells and three offset (staggered) lightemitting elements 74 extending across the height of the row, four cellsand four offset light emitting elements 74 extending across the heightof the row, five cells and five offset light emitting elements 74extending across the height of the row, more than five cells, two cells,etc.)

Each cell 76 has electrical (physical) pitch 116. Electrical pitch 116may accommodate routing components used to control the operation oflight emitting elements 114. In general, it may be desirable for pixelsize 110 to be relatively small. A larger electrical pitch 116 may allowsufficient space to accommodate electrical routing, thermal cooling,current density optimization, and contact resistance for the relativelysmall light emitting elements 114. Electrical pitch 116 may be, forexample, between 30 and 50 microns, between 10 and 50 microns, between35 and 45 microns, approximately 40 microns, etc. (e.g., in scenarioswhere light elements 74 are formed using VCSELs).

Control circuitry 16 may selectively activate different sets of lightemitting elements 74 in array 60 (e.g., using selected timing delaysacross the rows and columns of the array) while simultaneouslycontrolling the rotational frequency of scanning mirror 42 to displayany desired high resolution image frame in the far-field (e.g., bysynchronizing the timing/delays of pixel activation with the scanningmirror frequency and utilizing the large height of the 1.5D arrays tocover the second dimension (tangential axis) of the image frame). Byphysically staggering light emitting elements 74, controlling the timingof light emitting elements 74, and controlling scanning mirror 42 inthis way, light emitting elements 74 in array 60 may appear as a singlecontinuous column of light emitting elements (e.g., elements 74 mayproduce visually continuous image data that forms the displayed image)in the optical domain after reflection by scanning mirror 42 (e.g., asshown by diagram 123). This may allow light emitting elements 74 in the1.5D array, when combined with sagittal axis scanning by mirror 42, tofill out (paint) a high resolution, high frame-rate, two-dimensionalimage frame in the far-field (optical) domain (e.g., at other opticalcomponents 14C of FIG. 2). The optical domain 123 of the image frame mayexhibit an ultra-fine optical resolution (e.g., greater than 2048×1080resolution with a 3-6 micron effective pixel pitch) despite a coarserphysical (electrical) pitch 116 on arrays 60 (e.g., for accommodatingsignal routing, thermal dissipation, etc.).

If desired, control circuitry 16 may dynamically change the brightnessof different subsets of the light elements 74 in array 60 (e.g., usingpulse width modulation schemes, by adjusting current supplied to thelight emitting elements, by adjusting the source duty cycle, etc.). Thismay allow control circuitry 16 to locally brighten some or all of thedisplayed far-field image as needed (e.g., so that relatively brightimages are displayed when the user is in a bright room and so thatrelatively dim images are displayed to conserve power when the user isin a dark room). The operation of array 60 and scanning mirror 42 allowslight emitting elements 74 to be turned on most of the time, therebymaximizing the average brightness and visibility of the displayedimages. Display system 14 may support a brightness of thousands of nits,for example.

FIG. 7 is a diagram of an illustrative foveated image frame that may bedisplayed by display 14 (e.g., in the optical domain). As shown in FIG.7, image frame 132 may include a high definition foveated region 134 atthe center of the frame. Region 134 is surrounded by a lower resolutionperipheral region 130. Region 134 may be produced using high resolutionarrays 60 (FIG. 5) whereas region 130 is produced using lower resolutionarrays 62, for example. Generating an image frame of this type usingdisplay system 14 may allow display system 14 to conserve resourceswhile still providing image frames that appear natural to a viewer dueto perceived blurring at the periphery of the user's field of view.

FIG. 8 is a front view of projector 70 and components 68 of FIG. 5showing how routing circuitry and light source powering circuitry may beinterposed between the arrays of light sources 14A. As shown in FIG. 8,routing (interconnect) circuitry 100 may extend between arrays 60, 62,64, and 66 in projector 70. Routing circuitry 100 may convey controlsignals that turn the light emitting elements in the arrays on or off,that adjust the brightness of the light emitting elements, and/or thatgather sensor signals using sensor components in the arrays (e.g.,infrared photodiodes in array 66). Control circuitry 102 may also belocated between arrays 62 or between other arrays in components 14A.Control circuitry 102 may include synchronization and power integratedcircuits (e.g., application specific integrated circuits) and any otherdesired circuitry associated with powering or controlling components14A. The example of FIG. 8 is merely illustrative and, in general,routing circuitry and control circuitry may be mounted at any otherdesired locations.

If desired, microlenses may be provided over each light emitting elementin components 14A for directing the light emitted by the light emittingelements. FIG. 9 is a front view of a given array 60 that is providedwith microlenses. As shown in FIG. 9, microlenses such as microlenses140 may be located (e.g., centered) over each light emitting element 74in array 60 (e.g., each element 74 may have a corresponding microlens140). Like elements 74, microlenses 140 are vertically staggeredcolumn-to-column.

Each cell 76 in array 60 may be coupled to driver circuitry such as adriver integrated circuit over a corresponding contact pad. Whilemicrolenses 140 are aligned with light emitting elements 74, each cell76 in array 60 may include a contact pad that is centered with respectto the physical area of the cell. FIG. 10 is a rear view of a givenarray 60 that is provided with contact pads for coupling to a driverintegrated circuit. As shown in FIG. 10, each cell 76 in array 60includes a corresponding contact pad 150 that is centered with respectto that cell 76. Contact pads 150 may convey drive signals toselectively activate (or deactivate) the light emitting element 74within the corresponding cell. Unlike microlenses 140 of FIG. 9, contactpads 150 are not vertically staggered column-to-column (e.g., tosimplify packaging with the underlying driver integrated circuit).

FIG. 11 is a side view showing how two arrays 60 such as a first array60-1 and a second array 60-2 may be coupled to an underlying driverintegrated circuit. As shown in FIG. 11, contact pads 150 on the bottomsurface of substrate 75 in first array 60-1 are coupled to correspondingcontact pads 160 on driver integrated circuit 162 (e.g., an applicationspecific integrated circuit such as an active-matrix driver integratedcircuit). Similarly, contact pads 150 on the bottom surface of substrate75 of second array 60-2 are coupled to corresponding contact pads 160 ondriver integrated circuit 162. Contact pads 150 may be connected tocontact pads 160 using solder balls, a ball grid array, or any otherdesired conductive interconnect structures. Collectively, arrays 60 andintegrated circuit 162 may form a display (light source) integratedcircuit package 163.

If desired, display system 14 may include two scanning mirrors fordisplaying images. FIG. 12 is a top-down view of display system 14 in anexample where display system 14 includes two scanning mirrors such as afirst scanning mirror 224 and a second scanning mirror 226. As shown inFIG. 12, light sources 14A include multiple sets (groups) of laser dies204 each coupled to a corresponding laser driver 202. For example, lightsources 14A may include a first set of laser dies 204 coupled to andcontrolled by driver circuit 202A, a second set of laser dies 204coupled to and controlled by driver circuit 202B, and a third set oflaser dies 204 coupled to and controlled by driver circuit 202C. Thelaser dies 204 in each set may produce light of different wavelengths sothat each driver circuit 202 contributes light of a particular color todisplay system 14. Laser dies 204 may, for example, includeedge-emitting laser dies.

As shown in FIG. 12, optical system 14B may include coupling lenses 206,photonic integrated circuit (PIC) 200, one or more collimating optics222 (e.g., one or more collimating lenses), first scanning mirror 224,and second scanning mirror 226. Photonic integrated circuit 200(sometimes referred to herein as photonic light wave circuit (PLC) 200)may include a substrate 208 such as a glass substrate. Photolithographytechniques may be used to produce photolithographic patterns (channels)218 in substrate 208. Channels 218 may have a different index ofrefraction than the surrounding material in substrate 208, for example.

Each coupling lens 206 may be used to couple the light emitted from acorresponding laser die 204 into a respective one of channels 218. Forexample, lenses 206 may couple the light emitted by the laser dies 204coupled to driver 202A into channels 218 at side 214 of PIC 200, lenses206 may couple the light emitted by the laser dies 204 coupled to driver202B into channels 218 at side 216 of PIC 200, and lenses 206 may couplethe light emitted by the lasers 204 coupled to driver 202C into channels218 at side 212 of PIC 200.

Channels 218 have a relatively large pitch 236 at sides 214, 216, and212 of PIC 200 to accommodate the large size of laser dies 204. However,in order to maximize the resolution of the displayed image, channels 218may guide the light through PIC 200 to edge 210, where channels 218exhibit a reduced pitch 238. Pitch 238 may be, for example, between 10microns and 20 microns, approximately 12 microns, between 8 microns and16 microns, less than 8 microns, greater than 20 microns, between 5microns and 20 microns, etc. Channels 218 may convey the light fromlaser dies 204 through PIC 200 to surface 210 (e.g., via total internalreflection), where the light is emitted from PIC 200 as light 220.

Light fields emitted by PIC 200 may propagate to scanning mirror 224 viacollimating optics 222. Scanning mirror 224 may scan (rotate) aroundaxis 228 (as shown by arrows 230) at a relatively fast frequency (e.g.,across the sagittal axis of the displayed image). Scanning mirror 224may therefore sometimes be referred to herein as fast scanning mirror224 or fast mirror 224, and the sagittal axis may sometimes be referredto herein as the fast scan axis or the fast axis. Scanning mirror 224reflects light 220 towards scanning mirror 226. Scanning mirror 226 mayscan (rotate) around axis 232 (as shown by arrows 234) at a relativelyslow (low) frequency (e.g., across the tangential axis of the displayedimage). Scanning mirror 226 may therefore sometimes be referred toherein as slow scanning mirror 226 or slow mirror 226, and thetangential axis may sometimes be referred to herein as the slow scanaxis or the slow axis. Axes 228 and 232 may be orthogonal (e.g., becausethe tangential and sagittal axes of the display are orthogonal). Byscanning both mirrors 226 and 224 simultaneously, light 220 may be sweptacross both the tangential and sagittal axes of the display image tofill out (paint) a two dimensional image in the far-field (e.g., atother optical components 14C).

The light from the laser dies 204 coupled to driver 202A may bespatially offset from the light from the laser dies 204 coupled todriver 202B, which may be spatially offset from the light from the laserdies 204 coupled to driver 202C at edge 210 of PIC 200 (e.g., due to thepitch of channels 218 in PIC 200). This allows three distinct beams oflight of different colors (e.g., from each of drivers 202A, 202B, and202C) to be spatially offset at each of mirrors 224 and 226 and thusduring scanning across the displayed far-field image (e.g., with anangular separation between beams of 1-10 mRad relative to the tangentialaxis dimension). This may fill out more of the image with light relativeto scenarios where only a single laser is used, thereby maximizing theresolution of the displayed far-field image. The relatively fine pitch238 of channels 218 at edge 210 of PIC 200 further maximizes theresolution of the displayed far-field (e.g., to at least 1920×1080).Scanning mirrors 224 and 226 may rotate rapidly enough to supportrelatively high frame rates (e.g., 90 Hz or greater).

Concurrently scanning different beams of light (e.g., from each driver202) may also allow scanning mirror 224 to rotate more slowly whilecovering the sagittal axis than in scenarios where only a single beam isswept (e.g., because each beam will fill in some of the displayedfar-field image that would otherwise have to be covered using additionalcycles of a single beam). For example, scanning mirror 224 may rotate at20 kHz or lower whereas the scanning mirror would need to rotate at 20kHz or higher in scenarios where only a single beam is scanned. Thisreduction in rotational frequency for scanning mirror 224 may reducemechanical deformations in mirror 224 during rotation, therebyminimizing beam divergence and maximizing image resolution in thefar-field. Mirror 224 may also be up to 40% larger than the 27 kHzscanning mirror (e.g., without introducing excessive rotationaldeformation), further allowing mirror 224 to maximize far-field imageresolution. Mirror 226 may rotate at any desired frequency that is lessthan the rotational frequency of mirror 224.

The example of FIG. 12 is merely illustrative. In general, two drivercircuits 202 or more than three driver circuits 202 may be used. Eachdriver circuit 202 may drive any desired number of laser dies. Ifdesired, drivers 202, laser dies 204, optics 206, and/or PIC 200 may bemounted to a common substrate to form a single integrated opticalsystem, package, or integrated circuit 240.

FIG. 13 is a perspective view showing how lenses 206 may couple lightfrom laser dies 204 into PIC 200. As shown in FIG. 13, laser dies 204may include edge-emitting laser dies that emit light. Coupling lenses206 may include one or more ball lenses, one or more confocal lenses, asingle lens, a microlens mounted to laser die 204, or directly on edge216 of PIC 200, and/or other optical components to direct light fromlaser dies 204 onto or to otherwise couple the light into channels 218in PIC 200. Optical components 206 may be omitted and laser dies 204 mayemit light directly into channels 218 if desired.

If desired, the laser dies in display system 14 may be coupled to PIC200 over optical fibers. FIG. 14 is a top-down view showing how thelaser dies in display system 14 may be coupled to PIC 200 over opticalfibers. As shown in FIG. 14, light sources 14A may include RGB laserdies 270A, 270B, and 270C that each emit laser light of a correspondingcolor. RGB laser die 270A may be coupled to PIC 200 over optical fiber268A, RGB laser die 270B may be coupled to PIC 200 over optical fiber268B, and RGB laser die 270C may be coupled to PIC 200 over opticalfiber 268C. Optical fibers 268A, 268B, and 268C may be coupled to groovearray 262 on PIC 200. Groove array 262 may include grooves that hold theoptical fibers in place at fixed pitch 264. Pitch 264 is relativelylarge due to the physical size of the optical fibers. PIC die 200 mayinclude a pitch reduction region 260 that includes correspondingchannels 218 coupled to each optical fiber in groove array 262. Channels218 may exhibit pitch 264 at the boundary between regions 262 and 260and may exhibit a reduced pitch 266 at edge 210 of PIC 200 (e.g.,between 10 microns and 20 microns, approximately 12 microns, between 8microns and 16 microns, less than 8 microns, greater than 20 microns,between 5 and 20 microns, etc.). Channels 218 may subsequently emitseparate beams of light from RGB laser dies 270A, 270B, and 270C towardscollimator 222. Mirrors 224 and 226 may perform tangential and sagittalaxis scanning to fill in the two-dimensional far field image frame withthe beams of light emitted by laser dies 270A, 270B, and 270C.

The example of FIG. 14 is merely illustrative. In general, any desirednumber of laser dies 270 may be used (e.g., three or more laser dies 270or four or more laser dies 270 that emit light of any desired visible,ultraviolet, or infrared wavelengths). Laser dies 270, optical fibers268, and PIC 200 may be mounted to a common substrate to form a singleintegrated optical system, package, or integrated circuit 240 ifdesired.

The example of FIGS. 12-14 is merely illustrative. In general, light ofany desired wavelengths may be transmitted by PIC 200. If desired,wavelength multiplexing may be used in which different holographicelements in other optical components 14C perform different operations ondifferent wavelengths of red light, different wavelengths of greenlight, and different wavelengths of blue light, for example. PIC 200 maytransmit ultraviolet light and/or infrared light if desired. PIC 200 mayalso provide reflected infrared light to photodiodes coupled to PIC 200(e.g., to perform gaze tracking and/or depth sensing operations). Laserdies 204 and 270 need not include edge-emitting laser dies and may, ifdesired, include VCSELs or may be formed using any other desired lightemitting elements (e.g., LEDs). Drivers 202 may be coupled to dies 204(FIG. 12) using wire bonding, through silicon vias, ceramics or siliconinterposers, or any other desired interconnect structures. Substrate 208of PIC 200 may include glass, oxynitride, nitride, or any other desiredmaterials, may be formed in a single layer, or may include multiplelayers of material. PIC 200 may serve to reduce the relatively largepitch associated with light sources 204 and 270 to maximize theresolution of the displayed far-field image despite the use ofphysically large light sources. While FIGS. 12 and 14 illustrate twomirrors 224 and 226 as separate discrete mirrors (e.g., for performingrespective fast and slow axis scanning), this is merely illustrative andin another suitable arrangement, mirrors 224 and 226 may be implementedas a single mirror that is scanned over two degrees of freedom (e.g.,the fast (sagittal) and slow (tangential) axes, where each degree offreedom is scanned at the same frequency as a respective one of mirrors224 and 226 would be scanned in a scenario where two discrete mirrorsare used).

FIG. 15 illustrates plots showing how mirrors 224 and 226 (or a singlemirror in scenarios where the mirror is scanned over two degrees offreedom) may perform fast and slow scanning to fill a high resolutionimage frame in the far-field (optical domain). As shown in FIG. 15,graph 300 plots the scanning of an optical system that includes only asingle RGB light source. Curve 304 of graph 300 illustrates how the beamof light is swept across the two-dimensional image frame in thisscenario. As shown by curve 304, there is a relatively large amount ofempty (blank) space between each cycle of the beam. This serves to limitthe overall resolution of the displayed image.

Graph 302 of FIG. 15 plots the scanning of optical system 14B of FIGS.12-14. Curve 306 plots the beam produced by driver 202A of FIG. 12 ordie 270A of FIG. 14 (e.g., a beam of light of a first color). Curve 308plots the beam produced by driver 202B of FIG. 12 or die 270B of FIG. 14(e.g., a beam of light of a second color). Curve 310 plots the beamproduced by driver 202C of FIG. 12 or die 270C of FIG. 14 (e.g., a beamof light of a second color). Sweeping along the fast (sagittal) axis ofgraph 302 is performed by scanning mirror 224 whereas sweeping along theslow (tangential) axis of graph 302 is performed by scanning mirror 226.The pitch reduction provided by PIC 200 creates a very fine angularseparation between each beam (e.g., between 1-10 mRad relative to thetangential axis). This allows beams 306, 308, and 310 to fill in whatwould otherwise be empty (blank) space between a single beam (e.g., asshown by curve 304 of graph 300). This in turn minimizes the requiredscanning speed for mirror 224 (e.g., to 20 kHz or lower), allowing forthe diameter of mirror 224 to be maximized without incurringrotation-related deformation, and maximizes the resolution of thedisplayed two-dimensional far-field image. If desired, mirror 224 mayscan at a speed higher than 20 kHz (e.g., so that the mirror scansoutside of the audible range of the human ear).

If desired, the light in channels 218 of FIGS. 12-14 may be combined(coupled) between channels 218 using an optical combiner in PIC 200. Forexample, light of a first color may propagate through channel 218, lightof a second color may propagate through a second channel 218, and lightof a third color may propagate through a second channel 218 of PIC 200.A light combiner may include one or more optical couplers. The opticalcouplers may include portions of channels 218 that are brought intoclose proximity to each other so that light of different colors leaksbetween the channels. These types of optical couplers may be used toprovide any mixture of light of different colors to a given channel foroutput from the PIC. In this way, light of any desired colors may beemitted by any given channel 218 at edge 210 of PIC 200.

A physical environment refers to a physical world that people can senseand/or interact with without aid of electronic systems. Physicalenvironments, such as a physical park, include physical articles, suchas physical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

In contrast, a computer-generated reality (CGR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic system (e.g., an electronic systemincluding the display systems described herein). In CGR, a subset of aperson's physical motions, or representations thereof, are tracked, and,in response, one or more characteristics of one or more virtual objectssimulated in the CGR environment are adjusted in a manner that comportswith at least one law of physics. For example, a CGR system may detect aperson's head turning and, in response, adjust graphical content and anacoustic field presented to the person in a manner similar to how suchviews and sounds would change in a physical environment. In somesituations (e.g., for accessibility reasons), adjustments tocharacteristic(s) of virtual object(s) in a CGR environment may be madein response to representations of physical motions (e.g., vocalcommands).

A person may sense and/or interact with a CGR object using any one oftheir senses, including sight, sound, touch, taste, and smell. Forexample, a person may sense and/or interact with audio objects thatcreate 3D or spatial audio environment that provides the perception ofpoint audio sources in 3D space. In another example, audio objects mayenable audio transparency, which selectively incorporates ambient soundsfrom the physical environment with or without computer-generated audio.In some CGR environments, a person may sense and/or interact only withaudio objects. Examples of CGR include virtual reality and mixedreality.

A virtual reality (VR) environment refers to a simulated environmentthat is designed to be based entirely on computer-generated sensoryinputs for one or more senses. A VR environment comprises a plurality ofvirtual objects with which a person may sense and/or interact. Forexample, computer-generated imagery of trees, buildings, and avatarsrepresenting people are examples of virtual objects. A person may senseand/or interact with virtual objects in the VR environment through asimulation of the person's presence within the computer-generatedenvironment, and/or through a simulation of a subset of the person'sphysical movements within the computer-generated environment.

In contrast to a VR environment, which is designed to be based entirelyon computer-generated sensory inputs, a mixed reality (MR) environmentrefers to a simulated environment that is designed to incorporatesensory inputs from the physical environment, or a representationthereof, in addition to including computer-generated sensory inputs(e.g., virtual objects). On a virtuality continuum, a mixed realityenvironment is anywhere between, but not including, a wholly physicalenvironment at one end and virtual reality environment at the other end.

In some MR environments, computer-generated sensory inputs may respondto changes in sensory inputs from the physical environment. Also, someelectronic systems for presenting an MR environment may track locationand/or orientation with respect to the physical environment to enablevirtual objects to interact with real objects (that is, physicalarticles from the physical environment or representations thereof). Forexample, a system may account for movements so that a virtual treeappears stationery with respect to the physical ground. Examples ofmixed realities include augmented reality and augmented virtuality.

An augmented reality (AR) environment refers to a simulated environmentin which one or more virtual objects are superimposed over a physicalenvironment, or a representation thereof. For example, an electronicsystem for presenting an AR environment may have a transparent ortranslucent display through which a person may directly view thephysical environment. The system may be configured to present virtualobjects on the transparent or translucent display, so that a person,using the system, perceives the virtual objects superimposed over thephysical environment. Alternatively, a system may have an opaque displayand one or more imaging sensors that capture images or video of thephysical environment, which are representations of the physicalenvironment. The system composites the images or video with virtualobjects, and presents the composition on the opaque display. A person,using the system, indirectly views the physical environment by way ofthe images or video of the physical environment, and perceives thevirtual objects superimposed over the physical environment. As usedherein, a video of the physical environment shown on an opaque displayis called “pass-through video,” meaning a system uses one or more imagesensor(s) to capture images of the physical environment, and uses thoseimages in presenting the AR environment on the opaque display. Furtheralternatively, a system may have a projection system that projectsvirtual objects into the physical environment, for example, as ahologram or on a physical surface, so that a person, using the system,perceives the virtual objects superimposed over the physicalenvironment.

An augmented reality environment also refers to a simulated environmentin which a representation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

An augmented virtuality (AV) environment refers to a simulatedenvironment in which a virtual or computer generated environmentincorporates one or more sensory inputs from the physical environment.The sensory inputs may be representations of one or more characteristicsof the physical environment. For example, an AV park may have virtualtrees and virtual buildings, but people with faces photorealisticallyreproduced from images taken of physical people. As another example, avirtual object may adopt a shape or color of a physical article imagedby one or more imaging sensors. As a further example, a virtual objectmay adopt shadows consistent with the position of the sun in thephysical environment.

There are many different types of electronic systems that enable aperson to sense and/or interact with various CGR environments. Examplesinclude head mounted systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,uLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface. Thedisplay systems described herein may be used for these types of systemsand for any other desired display arrangements.

As described above, one aspect of the present technology is thegathering and use of data available from various sources to improve thedelivery of images to users, perform gaze tracking operations, and/or toperform other display-related operations. The present disclosurecontemplates that in some instances, this gathered data may includepersonal information data that uniquely identifies or can be used tocontact or locate a specific person. Such personal information data caninclude demographic data, location-based data, telephone numbers, emailaddresses, twitter ID's, home addresses, data or records relating to auser's health or level of fitness (e.g., vital signs measurements,medication information, exercise information), date of birth, or anyother identifying or personal information.

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used totrack a user's gaze to update displayed images and/or to perform otherdesired display operations. Accordingly, use of such personalinformation data enables users to view updated display images. Further,other uses for personal information data that benefit the user are alsocontemplated by the present disclosure. For instance, health and fitnessdata may be used to provide insights into a user's general wellness, ormay be used as positive feedback to individuals using technology topursue wellness goals.

The present disclosure contemplates that the entities responsible forthe collection, analysis, disclosure, transfer, storage, or other use ofsuch personal information data will comply with well-established privacypolicies and/or privacy practices. In particular, such entities shouldimplement and consistently use privacy policies and practices that aregenerally recognized as meeting or exceeding industry or governmentalrequirements for maintaining personal information data private andsecure. Such policies should be easily accessible by users, and shouldbe updated as the collection and/or use of data changes. Personalinformation from users should be collected for legitimate and reasonableuses of the entity and not shared or sold outside of those legitimateuses. Further, such collection/sharing should occur after receiving theinformed consent of the users. Additionally, such entities shouldconsider taking any needed steps for safeguarding and securing access tosuch personal information data and ensuring that others with access tothe personal information data adhere to their privacy policies andprocedures. Further, such entities can subject themselves to evaluationby third parties to certify their adherence to widely accepted privacypolicies and practices. In addition, policies and practices should beadapted for the particular types of personal information data beingcollected and/or accessed and adapted to applicable laws and standards,including jurisdiction-specific considerations. For instance, in the US,collection of or access to certain health data may be governed byfederal and/or state laws, such as the Health Insurance Portability andAccountability Act (HIPAA); whereas health data in other countries maybe subject to other regulations and policies and should be handledaccordingly. Hence different privacy practices should be maintained fordifferent personal data types in each country.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, in the caseof gaze tracking, the present technology can be configured to allowusers to select to “opt in” or “opt out” of participation in thecollection of personal information data during registration for servicesor anytime thereafter. In another example, users can select not toperform gaze tracking or other operations that gather personalinformation data. In yet another example, users can select to limit thelength of time gaze tracking is performed. In addition to providing “optin” and “opt out” options, the present disclosure contemplates providingnotifications relating to the access or use of personal information. Forinstance, a user may be notified upon downloading an app that theirpersonal information data will be accessed and then reminded again justbefore personal information data is accessed by the app.

Moreover, it is the intent of the present disclosure that personalinformation data should be managed and handled in a way to minimizerisks of unintentional or unauthorized access or use. Risk can beminimized by limiting the collection of data and deleting data once itis no longer needed. In addition, and when applicable, including incertain health related applications, data de-identification can be usedto protect a user's privacy. De-identification may be facilitated, whenappropriate, by removing specific identifiers (e.g., date of birth,etc.), controlling the amount or specificity of data stored (e.g.,collecting location data a city level rather than at an address level),controlling how data is stored (e.g., aggregating data across users),and/or other methods.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, display imagesbased on non-personal information data or a bare minimum amount ofpersonal information, such as the content being requested by the deviceassociated with a user, other non-personal information available to thedisplay system, or publicly available information.

The foregoing is merely illustrative and various modifications can bemade to the described embodiments. The foregoing embodiments may beimplemented individually or in any combination.

What is claimed is:
 1. A display system configured to display an imageframe, the display system comprising: first, second, and third sets oflaser dies, wherein the first set of laser dies is configured to emit afirst set of beams of light, the second set of laser dies is configuredto emit a second set of beams of light, and the third set of laser diesis configured to emit a third set of beams of light, wherein the first,second, and third sets of beams each include light in at least threewavelength ranges; a photonic integrated circuit, wherein the first,second, and third sets of laser dies are configured to respectively emitthe first, second, and third sets of beams into the photonic integratedcircuit with a first non-zero pitch, and wherein the photonic integratedcircuit is configured to emit the first, second, and third set of beamswith a second non-zero pitch that is finer than the first non-zeropitch; a first scanning mirror configured to reflect the first, second,and third sets of beams while scanning along a first axis of the imageframe; and a second scanning mirror configured to reflect the first,second, and third sets of beams while scanning along a second axis ofthe image frame.
 2. The display system defined in claim 1, wherein thefirst axis comprises a sagittal axis and the first scanning mirror isconfigured to scan along the sagittal axis at a first frequency.
 3. Thedisplay system defined in claim 2, wherein the second axis comprises atangential axis and the second scanning mirror is configured to scanalong the tangential axis at a second frequency that is less than thefirst frequency.
 4. The display system defined in claim 3, wherein thefirst frequency is greater than 20 kHz.
 5. The display system defined inclaim 3, further comprising output collimating optics interposed betweenthe photonic integrated circuit and the first scanning mirror that areconfigured to collimate the first, second, and third sets of beams andto direct the first, second, and third sets of beams at the firstscanning mirror, the first scanning mirror being configured to thenreflect the first, second, and third sets of beams towards the secondscanning mirror.
 6. The display system defined in claim 3, wherein theat least three wavelength ranges comprise first, second, and thirdvisible colors.
 7. The display system defined in claim 1, wherein thephotonic integrated circuit comprises a substrate having a first indexof refraction and a plurality of channels in the substrate having asecond index of refraction that is different from the first index ofrefraction, the plurality of channels being configured to convey thefirst, second, and third sets of beams and to transmit the first,second, and third beams sets of beams out of the photonic integratedcircuit.
 8. The display system defined in claim 1 wherein, upon emissionby the photonic integrated circuit: the first set of beams is spatiallyoffset with respect to the second set of beams, and the third set ofbeams is spatially offset with respect to the first set of beams andwith respect to the second sets of beams.
 9. The display system definedin claim 1, further comprising: coupling lenses configured to couple thefirst, second, and third sets of beams into the photonic integratedcircuit.
 10. The display system defined in claim 1, further comprising:a first laser driver coupled to each laser die in the first set of laserdies; a second laser driver coupled to each laser die in the second setof laser dies; and a third laser driver coupled to each laser die in thethird set of laser dies.
 11. The display system defined in claim 10,wherein the first, second, and third sets of laser dies each compriseedge-emitting laser dies.
 12. The display system defined in claim 1,further comprising: a first set of optical fibers that couples the firstset of laser dies to the photonic integrated circuit; a second set ofoptical fibers that couples the second set of laser dies to the photonicintegrated circuit; and a third set of optical fibers that couples thethird set of laser dies to the photonic integrated circuit.
 13. Thedisplay system defined in claim 12, wherein the photonic integratedcircuit comprises a groove array that holds the first, second, and thirdsets of optical fibers in place.
 14. Apparatus comprising: a first setof light sources configured to emit a first set of beams in at leastthree wavelength ranges; a second set of light sources configured toemit a second set of beams in at least three wavelength ranges; aphotonic integrated circuit having opposing first and second edges andhaving a lateral surface that extends from the first edge to the secondedge, the photonic integrated circuit comprising: a first set ofchannels configured to receive the first set of beams through the firstedge of the photonic integrated circuit and configured to transmit thefirst set of beams out of the second edge of the photonic integratedcircuit, and a second set of channels configured to convey the secondset of beams and configured to transmit the second set of beams out ofthe second edge of the photonic integrated circuit; a sagittal axisscanning mirror configured to reflect the first and second sets ofbeams; and a tangential axis scanning mirror configured to reflect thefirst and second sets of beams reflected by the sagittal axis scanningmirror.
 15. The apparatus defined in claim 14, wherein the first set ofbeams are separated from each other and the second set of beams areseparated from each other upon transmission from the photonic integratedcircuit by a corresponding angular separation and wherein the angularseparation is between 1 mRAD and 10 mRAD.
 16. The apparatus defined inclaim 14, wherein each channel in the first set of channels is separatefrom each other channel in the first set of channels and wherein eachchannel in the second set of channels is separate from each otherchannel in the first and second sets of channels.
 17. The apparatusdefined in claim 14, wherein the photonic integrated circuit hasopposing third and fourth edges, the lateral surface extends from thethird edge to the fourth edge, and the second set of channels isconfigured to receive the second set of beams through the third edge ofthe photonic integrated circuit.
 18. The apparatus defined in claim 14,wherein the first and second sets of light sources comprise a lightsource selected from the group consisting of: an edge-emitting laserdie, a vertical cavity surface emitting laser (VCSEL), a resonant cavitylight emitting diode (RCLED), and a light emitting diode (LED).
 19. Theapparatus defined in claim 14, further comprising: a third set of lightsources configured to emit a third set of beams in at least threewavelength ranges; a third set of channels in the photonic integratedcircuit configured to convey the third set of beams and to transmit thethird set of beams out of the second edge of the photonic integratedcircuit; and a light coupler configured to couple at least some of thethird set of beams into the second set of channels.
 20. The apparatusdefined in claim 14, wherein the at least three wavelength rangescomprise at least one visible wavelength range.
 21. The apparatusdefined in claim 20, wherein the at least three wavelength rangescomprise at least one infrared wavelength range.
 22. The apparatusdefined in claim 14, wherein the photonic integrated circuit hasopposing third and fourth edges, the lateral surface extends from thethird edge to the fourth edge, and the second set of channels isconfigured to receive the second set of beams through the first edge ofthe photonic integrated circuit.
 23. The apparatus defined in claim 22,wherein the photonic integrated circuit has an additional lateralsurface opposite the lateral surface, the apparatus further comprising:a substrate, wherein the additional lateral surface of the photonicintegrated circuit is mounted to a surface of the substrate; couplinglenses mounted to the surface of the substrate and configured to directthe first and second sets of beams towards the first edge of thephotonic integrated circuit; a first driver mounted to the surface ofthe substrate and configured to drive the first set of light sources;and a second driver mounted to the surface of the substrate andconfigured to drive the second set of light sources, wherein the first,second, third, and fourth edges of the photonic integrated circuitextend perpendicular to the surface of the substrate, the lateralsurface of the photonic integrated circuit, and the additional lateralsurface of the photonic integrated circuit.
 24. Apparatus comprising: afirst set of laser dies driven by a first laser driver and configured toemit a first set of beams in at least three wavelength ranges; a secondset of laser dies driven by a second laser driver and configured to emita second set of beams in at least three wavelength ranges; and aphotonic integrated circuit (PIC) having first, second, third, andfourth edges, a lateral surface extending between the first, second,third, and fourth edges, and comprising a first set of channels and asecond set of channels, wherein: the first edge opposes the second edge,the third edge opposes the fourth edge, each channel in the first set ofchannels extends from the first edge of the PIC to the second edge ofthe PIC and is separate from each other channel in the first set ofchannels, each channel in the second set of channels extends from thethird edge of the PIC to the second edge of the PIC and is separate fromeach other channel in the first and second sets of channels, eachchannel in the first set of channels is configured to receive, throughthe first edge of the PIC, a respective beam of the first set of beamsemitted by the first set of laser dies, each channel in the second setof channels is configured to receive, through the third edge of the PIC,a respective beam of the second set of beams emitted by the second setof laser dies, the first set of channels has a first pitch at the firstedge, the second set of channels has the first pitch at the third edge,and the first and second sets of channels have a second pitch at thesecond edge of the PIC that is less than the first pitch.